Numerical Issues Involved in Inverting Hessian Matrices

In the social sciences, researchers typically assume the accuracy of generalized linear models by using an asymptotic normal approximation to the likelihood function or, occasionally, by using the full posterior distribution. Thus, for standard maximum likelihood analyses, only point estimates and the variance at the maximum are normally seen as necessary. For Bayesian posterior analysis, the maximum and variance provide a useful first approximation (but see Chapter 4 for an alternative). Unfortunately, although the negative of the Hessian (the matrix of second derivatives of the posterior with respect to the parameters and named for its inventor in slightly different context, German mathematician Ludwig Hesse) must be positive definite and hence invertible so as to compute the variance matrix, invertible Hessians do not exist for some combinations of datasets and models, so statistical procedures sometimes fail for this reason before completion. Indeed, receiving a computer-generated "Hessian not invertible" message (because of singularity or nonpositive definiteness) rather than a set of statistical results is a frustrating but common occurrence in applied quantitative research. It even occurs with regularity during many Monte Carlo experiments where the investigator is drawing data from a known statistical model, due to machine effects. The Hessian can be noninvertible for both computational reasons and data reasons. Inaccurate implementation of the likelihood function (see Chapters 2 and 3), inaccurate derivative methods (see Chapter 8), or other inappropriate choices in optimization algorithms can yield noninvertible Hessians. Where these inaccuracies cause problems with Hessians, we recommend addressing these inaccuracies directly. If these methods aren't feasible, or don't work, which often happens, we provide an innovative new library for doing generalized inverses . Moreover, when a Hessian is not invertible for data reasons, no computational trick can make it invertible, given the model and data chosen, because the desired inverse does